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ABSTRACT 

Nova’s  development  of  the  “Modular  Infrared  Imaging  Applications  Development  System”  (MIRIADS)  produced  a 
longwave  infrared  (LWIR)  camera  system  that  operated  the  “Adaptive  Infrared  Sensor”  (AIRS)  focal  plane  device  produced 
by  the  Raytheon  Infrared  Operations  (RIO)  organization.  A  novel  system  architecture  permitted  the  integration  of  an  infrared 
fisheye  lens  system  produced  by  Optics  1,  Inc.,  which  permitted  a  complete  hemispherical  field  of  view  to  be  imaged  onto 
the  AIRS  FPA.  This  paper  will  describe  applications  for  this  system  as  an  extremely  wide  field-of-view  IR  sensor  (early 
warning  detection,  fire  detection,  etc.),  and  will  present  test  imagery  collected  with  the  system. 

This  technology  advancement  has  been  the  result  of  the  coordinated  effort  of  a  variety  of  companies  and  government 
agencies.  This  presentation  will  highlight  significant  contributions  of  individuals  and  will  indicate  the  effectiveness  of  the 
Small  Business  Innovative  Research  (SBIR)  program  in  helping  to  advance  this  nation’s  technology  base. 

FPA,  LWIR,  AIRS,  electronics,  dewar  packaging,  fisheye  optics,  sensor  processing 


1.  INTRODUCTION 

Under  sponsorship  from  the  Air  Force  Research  Laboratory  Munitions  Directorate,  Nova  Research,  Inc.  has  successfully 
completed  the  development  of  the  MIRIADS  system,  an  infrared  camera  system  that  has  the  capability  to  operate  virtually  all 
existing  infrared  focal  plane  arrays,  with  the  additional  capability  to  perform  powerful  real-time  signal  and  image  processing 
on  the  resulting  image  data.  The  great  flexibility  of  the  system  architecture  will  speed  the  time  required  for  existing  FPA 
resources  to  be  used  in  imaging  tests,  and  the  programmable  nature  of  the  signal  processing  system  will  permit  a  variety  of 
mission- specific  algorithms  to  be  applied  to  the  real-time  digital  data. 

This  paper  presents  an  overview  of  initial  imaging  tests  using  the  MIRIADS  system  with  the  AIRS  FPA  and  the  Optics  1 
LWIR  fisheye  lens  system.  A  variety  of  data  has  now  been  collected  with  this  system  and  applications  for  this  data  will  be 
discussed  in  this  paper.  Aircraft,  automobiles,  watercraft  and  other  terrestrial  data  have  been  collected  and  a  variety  of 
system  applications  are  under  consideration. 

2.  DESCRIPTION  OF  THE  AIRS/FISHEYE  MIRIADS  CAMERA  SYSTEM 

As  pictured  in  Figure  2-1,  the  AIRS/Fisheye  MIRIADS  system  may  be  used  to  collect  a  wide  variety  of  LWIR  imagery.  The 
right  inset  of  the  figure  shows  the  single-board  “motherboard”  that  operates  the  FPA  in  its  cryogenic  dewar  structure.  In 
addition,  the  LWIR  fisheye  lens  is  mounted  to  the  front  of  the  dewar  system  with  a  focus  mechanism.  A  variety  of  signal 
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processing  boards  may  be  plugged  onto  the  backside  of  the  motherboard  for  application  of  special-purpose  processing 
operations. 


Used  in  conjunction 
with  RIO  “Adaptive 
Infrared  Sensor” 
(AIRS),  LWIR,  for  IR 
early-warning  sensor 
demonstration. 


Optics  1,  Inc.  Has  Developed  an 
LWIR  Fisheye  Lens  System 
With  Full  Hemispherical  FOV 


Figure  2-1.  An  initial  data  collection  effort  provided  a  variety  of  aircraft  and  terrestrial  data  sets. 

As  pictured  in  the  center  inset,  the  combined  weight  of  the  MIRIADS  camera  head  with  its  large-capacity  LN2  reservoir  filled 
with  LN2  is  approximately  nine  pounds.  The  large-capacity  LN2  reservoir  may  be  replaced  with  a  cryocooler  for  remote 
and/or  airborne  data  collection  purposes. 

A  design  goal  was  to  produce  a  system  that  provided  an  easy  interface  to  a  commercially-available  image  capture  and  display 
system  so  as  not  to  divert  from  the  main  goal  of  producing  a  highly  modular,  configurable  system.  For  this  reason,  the  SE-IR 
Corporation’s  “CamIRa”  display  frame  grabber,  DSP  board  and  software  was  identified  and  used  to  provide  real-time  display 
and  digital  data  storage  functions. 


3.  PRELIMINARY  IMAGE  DATA 

Two-point  Nonuniformity  Correction  (NUC)  is  a  difficult  operation  with  the  fisheye-based  system  operating  the  AIRS  in 
direct  injection  mode  because  the  signal  variation  across  the  image  plane  is  largely  due  to  the  spatial  distortion  of  the  lens. 
When  the  FPA’s  integration  time  is  increased  to  improve  the  sensitivity  of  the  system,  the  signal  “bow”  quickly  fills  the 
dynamic  range  of  the  sensor,  resulting  in  saturation  of  pixels  in  the  periphery  of  the  fisheye’ s  field  of  view. 

The  electronics  on  the  DSP  board  of  the  SE-IR  system  were  designed  to  perform  NUC  corrections  based  on  signal  variations 
that  do  not  exceed  approximately  50%  of  the  instantaneous  dynamic  range  (i.e.  50%  of  the  input  span  of  the  A/D  converter). 
Above  approximately  this  value,  the  resulting  NUC  gains  required  exceed  those  supported  by  the  NUC-processing  hardware. 

It  is  for  this  reason  that  the  temporal  highpass  filtering  (THPF)  operating  mode  of  the  AIRS  provides  a  great  advantage  when 
operating  the  system  using  the  fisheye  lens.  In  essence,  the  THPF  processing  will  have  the  result  of  performing  an  “analog 
pre-corrected”  offset  correction  on  the  FPA  itself,  thereby  permitting  the  SE-IR  DSP  hardware  to  produce  a  high  quality  2- 
point  correction. 


As  a  preliminary  effort  to  visualize  image  frames  from  the  LWIR  fisheye  system,  sequences  of  frames  were  collected  in  the 
laboratory  without  application  of  real-time  THPF  or  Subframe  Averaging.  Two-point  nonuniformity  corrections  were 
applied  using  a  software  approach.  A  representative  sequence  is  shown  in  Figure  3-1. 


Figure  3-1.  Well-corrected  frames  subsampled  from  a  100-frame  sequence,  produced  with  application  of  a  floating¬ 
point  two-point  NUC  routine. 

One  might  expect  that  applying  a  uniform  blackbody  illumination  profile  to  a  fisheye  lens-based  system  is  difficult.  We 
found  that  a  reasonably  uniform  illumination  source  was  the  hemispherical  inside  surface  of  a  metal  ice  cream  scoop  that  was 
spray -painted  with  flat  black  to  produce  a  good,  diffuse  surface.  One  scoop  was  used  at  room  temperature  for  the  “cold” 
scene,  and  the  other  was  dipped  into  a  cup  of  boiling  water,  removed  from  the  water  and  dried  with  a  heat  gun,  then  used  to 
expose  the  sensor  for  the  “hot”  scene. 


Total  system  noise  was  measured  by  computing  the  standard  deviation  of  a  group  of  3300  pixels  in  a  non-imaged  portion  of 
the  AIRS  FPA  as  indicated  in  Figure  3-2.  Five  separate  data  collection  cycles  were  used  to  measure  an  average  standard 
deviation  of  approximately  2.0  counts  on  a  14-bit  scale  (i.e.,  2  counts  out  of  a  total  dynamic  range  of  16,384  counts). 
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Figure  3-2.  Five  data  histograms  taken  from  a  non-imaged  portion  of  the  AIRS  FPA  show  an  average  standard 
deviation  of  approximately  2.0  counts  on  a  14-bit  scale  (16,384  total  counts). 


4.  REAL-TIME  TEMPORAL  HIGHPASS  FILTERING  MODES 


The  AIRS  has  been  designed  to  operate  in  three  primary  modes  of  operation.  The  first  is  a  standard  Direct  Injection  (DI) 
mode  whereby  the  FPA  operates  as  a  standard  2D  imager.  A  second  mode  of  operation  is  Subframe  Averaging.  The 
advantage  of  this  mode  is  that  by  averaging  subframes,  sensitivity  is  improved  by  reducing  the  effect  of  high  energy  photon 
shot  noise.  The  third  mode  of  operation,  which  we  will  focus  on  here,  is  Temporal  Highpass  Filtering  (THPF).  This  mode  is 
effective  in  reducing  fixed  pattern  noise  as  well  as  the  low  frequency  noise  and  drift  components  from  the  scene.  The 
number  of  pixels  operating  with  marginal  performance  is  improved.  As  mentioned  earlier,  the  THPF  processing  will  in 
effect  have  the  result  of  performing  an  “analog  pre-corrected”  offset  correction  on  the  FPA  itself  thus  permitting  the  SE-IR 
DSP  hardware  to  produce  a  high  quality  2-point  corrected  image.  Figure  4-1  shows  six  representative  frames  of  a  100-frame 
sequence  in  which  the  nonlinear  effects  of  fisheye-lens-induced  illumination  profile  variations  are  compensated  using  this 
process. 


Frame  60  Frame  80  Frame  100 


Figure  4-1.  Fixed  pattern  image  variation  due  to  fisheye  lens  illumination  profile  is  removed  using  a  Temporal 

Highpass  Filtering  mode. 

Frame  1  indicates  the  first  frame  in  a  representation  of  how  the  sensor  would  respond  in  a  direct  injection  mode  without  the 
compensating  effects  of  temporal  highpass  filtering.  This  is  the  first  frame  of  a  100-frame  sequence  in  which  the  THPF 
offset  compensation  is  correcting  (i.e.,  “flattening”)  the  response  of  the  sensor.  Subsequent  frames  spaced  at  twenty-frame 
intervals  are  shown.  Respective  plots  of  even-numbered  pixels  in  the  128th  row  (i.e.,  in  the  horizontal  center  row  of  each 
image)  are  given  in  Figure  4-2  for  reference.  The  temporal  noise  incorporated  in  this  example  is  representative  of  that 
achieved  in  the  high  background  imaging  condition  (F/1.0)  of  the  LWIR  fisheye  lens  operated  with  the  AIRS  LWIR  FPA. 
The  detector  operates  in  a  lower  noise  condition  when  the  illumination  flux  is  lower,  limited  by  an  optical  system  having  an 
F-number  of  approximately  2.3,  presumably  due  to  the  higher  dynamic  resistance  (RoA)  of  the  detector  at  lower  flux  levels. 

Applying  temporal  highpass  filtering  to  real-time  image  data  can  provide  a  valuable  tool  for  image  tracking  applications,  at 
the  expense  of  traditional  image  quality  parameters.  The  upper  three  frames  of  Figure  4.3  present  MIRIADS  LWIR  fisheye 
imagery  with  the  AIRS  FPA  operating  in  its  “Direct  Injection”  (DI)  mode  before  application  of  any  temporal  highpass 
filtering.  Notice  that  there  are  numerous  fixed  images  in  the  three  frames  (overhead  lighting  and  other  laboratory  thermal 
sources).  The  lower  three  frames  of  Figure  4-3  show  the  same  three  frames  that  have  now  been  processed  with  AIRS-like 
temporal  highpass  filtering. 


Figure  4-2.  Image  data  corresponding  to  the  images  shown  in  Figure  4-1  indicate  the  rate  at  which  fixed  pattern 
image  data  is  removed  during  THPF  processing.  Notice  that  the  desired  signal  flattening  behavior  is  shown,  and  that 
signal  fluctuation  due  to  temporal  noise  will  pass  through  the  process. 


Frame  94  THPF  Frame  96  THPF  Frame  98  THPF 

Figure  4-3.  Application  of  Temporal  Highpass  Filtering  to  actual  image  data  can  provide  useful  target  tracking 

information. 


Some  important  aspects  of  the  imagery  in  Figure  4-3  are  worth  noting: 


•  Spatial  fixed  pattern  imagery  has  been  reduced  in  intensity  or  completely  removed.  This  is  because  only  higher-speed 
objects  will  pass  through  the  THPF  process.  You  may  see  this  by  noting  that  the  overhead  lighting  fixtures  that  were 
visible  in  the  upper  (input)  frames  have  been  erased  in  the  lower  frames. 

•  The  moving  person  in  the  input  frames  remains  in  the  lower  frames  by  virtue  of  his  motion. 

•  Bright  sources  will  not  only  pass  through  to  the  output,  but  if  bright  enough,  will  produce  a  “latent”  dark  image  that  will 
fade  away  with  time  based  on  the  temporal  filtering  adjustments  provided  by  the  AIRS  FPA.  While  not  useful  from  a 
conventional  “image  quality”  sense,  the  ghost  images  record  a  time  history  of  the  motion  of  the  objects,  which  could  be 
of  great  use  in  target  tracking  applications.  As  an  example  of  this,  the  particularly  dark,  large  region  in  the  lower  left 
portion  of  the  lower  frames  was  produced  because  the  soldering  iron  happened  to  dwell  in  this  region  for  a  reasonably 
long  period  of  time  before  this  particular  image  frames  were  recorded. 


5.  POTENTIAL  APPLICATIONS 


Wide  Field  of  View  Aircraft  Sensor 

Nova  Research,  Inc.  has  had  a  long  and  fruitful  working  relationship  with  Tanner  Research,  Inc.,  in  which  Nova  has 
produced  system-level  infrared  sensor  hardware,  and  Tanner  has  developed  a  sensor  processing  algorithm,  the  Wave 
Process1,  designed  to  detect  low  contrast  single-pixel  moving  objects  immersed  in  cluttered  backgrounds. 

An  extension  to  the  development  contract  used  to  design  and  produce  the  MIRIADS  system  described  in  this  paper  is 
concentrating  on  a  “real-time”  application  of  Tanner’s  Wave  Process  algorithm  to  live  camera  data  produced  by  MIRIADS. 
In  preparation  for  such  high-speed  processing,  we  collected  through  the  fisheye  lens  a  variety  of  data  on  flying  aircraft,  and 
applied  the  Wave  Process  off-line  to  the  data  to  investigate  the  possibility  of  increased  probability  of  detection  for  such  low- 
contrast  targets.  These  targets  become  unresolved  “sub-pixel”  targets  when  using  the  fisheye  lens  because  the  instantaneous 
field  of  view  (IFOV)  of  a  pixel  subtends  a  relatively  large  solid  angle,  as  compared  to  the  case  in  which  a  conventional  lens  is 
used  to  image  over  a  much  narrower  total  field  of  view  (TFOV). 

Figure  5-1  shows  three  panels  of  frame  clips.  The  left  group  of  clips  in  each  panel  are  individual  frames  from  the  256  x  256 
pixel  AIRS  FPA  fisheye  lens  image.  As  the  sequence  proceeds,  an  aircraft  takes  off  from  left  to  right;  without  using  some 
technique  for  improving  the  contrast  of  the  aircraft  with  respect  to  background,  it  is  very  difficult  to  see  the  aircraft  target. 

The  right  group  of  clips  in  each  panel  are  the  respective  Wave-Processed  versions  of  these  frames.  Notice  a  characteristic 
chevron-shaped  “wake”  feature  that  is  produced  by  the  algorithm.  The  Wave  Process  improves  the  detectability  of  objects 
which  move  in  a  manner  that  is  correlated  in  space  and  time  by  aggregating  the  detected  signal  over  time,  producing  a 
wavefront  with  an  amplitude  that  grows  as  the  target  is  observed.  Other  moving  objects  in  the  field  of  view  (e.g.,  people  in 
the  foreground  in  the  right  portion  of  the  frame)  will  also  produce  such  wave-like  results  and  ongoing  research  at  Tanner 
Research  is  concentrating  on  preferentially  detecting  objects  of  interest. 

The  Wave  Process  may  be  applied  to  a  variety  of  applications  in  which  the  target  contrast  is  low  with  respect  to  its 
background.  Real-time  application  of  this  technique  at  Nova  is  through  the  use  of  MIRIADS-generated  digital  data 
interfaced  to  a  SKY  Computers,  Inc.  SKYsds-1  computer  system.  Numerous  speed-enhancing  operations  have  been 
incorporated  into  a  modified  version  of  the  Wave  Process;  Nova  expects  to  achieve  a  30  frame/second  processing  speed  of 
128  x  128  pixel  camera  data.  Similar  results  may  be  possible  with  other  sensor  types  including  sonar,  LADAR  imagers,  UV, 
and  passive  millimeter  wave  imagers. 


1  T.J.  Bartolac,  “Recent  Developments  in  the  Wave  Process,”  SPIE,  Smart  Focal  Plane  Arrays  and  Focal  Plane  Testing,  Vol. 
2474,  April,  1995,  pp.  40-50. 
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Figure  5-1.  The  AIRS/Fisheye  MIRIADS  was  used  to  collect  data  on  aircraft  taking  off  (left  panels  in  each  group), 
and  when  the  Wave  Process  was  applied  to  this  frame-based  data  the  detectability  of  the  aircraft  is  greatly  improved. 


Hemispherical  Field  of  View  Optical  Fire  Detector 

We  have  investigated  the  possibility  of  using  this  hemispherical  FOV  system  in  a  remote  fire-detection  application.  For 
example,  the  2n  steradian  field  of  view  characteristic  of  the  Optics  1  fisheye  lens  could  be  exploited  to  find  the  initial 
outbreak  of  open  flames  in  a  large  convention  hall.  A  single  sensor  located  in  the  ceiling  would  have  full  coverage  of  an 
entire  large  indoor  convention  hall  and  the  processing  described  below  could  be  applied  to  the  real-time  sensor  data  to  help 
identify  the  presence  of  the  fire  and  provide  the  resulting  information  to  fire  control  systems. 

As  a  simple  demonstration,  a  butane  lighter  was  placed  in  the  laboratory,  and  the  AIRS/Fisheye  MIRIADS  system  collected 
256  frames  of  image  data  at  a  speed  of  approximately  7.0  frames/second,  as  shown  in  Figure  5-2.  Higher  frame  rates  are 
possible  (up  to  the  sensor’s  operational  speed  of  approximately  60  frames/second). 


Temporal  Samples  of  Flame  Flicker 
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Figure  5-2.  A  single  pixel  in  the  LWIR  image  of  a  butane  flame  was  monitored  for  256  frames  in  preparation  for 
application  of  an  FFT  to  determine  component  flicker  frequencies. 

A  simple  data  analysis  routine  was  created  that  permits  the  user  to  specify  a  given  pixel  that  is  in  the  vicinity  of  the  flame’s 
time- varying  image,  and  the  256  consecutive  samples  were  recorded  and  subsequently  analyzed  using  an  FFT  algorithm. 
Figure  5-3  shows  the  resulting  characteristic  frequencies  of  the  flickering  flame.  This  small  dataset  taken  at  a  relatively  slow 
sample  rate  produces  a  result  that  is  band-limited  (resulting  in  a  false  indication  of  the  importance  of  higher  frequency 
components),  but  the  Power  Spectral  Density  (PSD)  plot  in  Figure  5-4  gives  a  reasonably  good  indication  of  the  importance 
of  such  relative  spectral  powers. 


Figure  5-3.  Disk  I/O  overhead  in  this  data  collection  was  limited  to  recording  at  7  frames/second.  This  resulting 
frequency  spectrum  was  produced,  showing  a  characteristic  flicker  peak  at  approximately  0.87  Hz. 
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Figure  5-4.  Relative  Power  Spectral  Densities  of  flame  flicker  indicates  a  characteristic  flicker  frequency  at  0.87  Hz. 


In  future  commercial  applications,  the  MIRIADS  system  could  be  configured  to  compute  a  real-time  FFT  similar  to  the  result 
shown  above,  and  compare  characteristic  frequencies  with  known  spectral  distribution  signatures  of  actual  fires.  In  so  doing, 
the  goal  would  be  to  reduce  the  false  alarm  rate  while  operating  the  system  with  full  FOV  coverage.  An  important 
advantage  of  this  system  would  be  to  communicate  this  information  to  fire  extinguishing  systems  that  would  apply  fire 
retardant  to  only  specific  regions  in  the  protected  zone  such  that  sensitive  equipment  in  non-fire  areas  would  not  suffer 
potentially  damaging  application  of  water  or  foam. 
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